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Introduction 

This paper introduces the Affordance Template framework 
used to supervise task behaviors on the NAS A- JSC Valkyrie 
robot at the 2013 DARPA Robotics Challenge (DRC) Trials. 
This framework provides graphical interfaces to human su- 
pervisors that are adjustable based on the run-time environ- 
mental context (e.g., size, location, and shape of objects that 
the robot must interact with, etc.). Additional improvements, 
described below, inject degrees of autonomy into instantia- 
tions of affordance templates at run-time in order to enable 
efficient human supervision of the robot for accomplishing 
tasks. 


The Affordance Template Framework 

Recent advances in human-robot interface design have led 
to a number of useful tools for interacting with complex sys- 
tems. Specifically, the abundance of RGBD and LIDAR de- 
vices have resulted in the development of 3D visualizations 
of robot environments that reflect state of the robot in a vir- 
tual space with the aggregate data delivered by these devices. 
One such environment is RViz, which enables seemless use 
with the Robot Operating System (ROS). While RViz deliv- 
ers a significant amount of off-the-shelf situational aware- 
ness to a human supervisor along with a stan dard point-and- 
click interface for teleoperation ( [Gossow et al. 201 1| , it does 
not inherently provide the tools for directing robots to ac- 
complish more complex tasks. Affordance templates, intro- 
duced here, have been developed to provide such task-level 
structure. 

An affordance describes a place or object in the world 
that affords an action by a particular agent. If an area of 
the perceived environment affords “sitting” it is not that 
that area is a chair, but rather that it has a surface that 
the agent could sit on (without falling off), whether it be 
a chair, a stool, a rock, or the ground, and is inextricably 
linked to the agent (robot) doing the acting. The concept 
of affordances was initially introduced in the psychologi- 
cal literature by Gibson ( Gibson 1977) as a means of de- 
scribing cognitive structures that assign functional merit to 
the environment. It is therefore readily applicable for pro- 
gramming robots; embodied agents that must perform ac- 
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Figure 1 : Placing a wheel- turning template in RViz. 


tions to achieve some discernible objective ( [Stoytchev 2005} 
|Hart and Grupen 2013| ). 

An affordance template is a graphical construct that exists 
in a 3D immersive environment (such as RViz) to provide 
human-adjustable robot task goals (spatial end-effector way- 
points, contact points for interaction, force magnitudes) and 
parameters (object scales, locations) in object-centric coor- 
dinate frames. If a template is placed in such an environment 
alongside a robot avatar and its aggregate sensory data, a su- 
pervisor can move the template to an appropriate location (a 
location that affords the task behavior) and adjust the tem- 
plate goals and parameters as necessary, appropriate for the 
current run-time context. 

Figure [T] shows the placement of a wheel-turning affor- 
dance template in the immersive RViz environment. In (a), 
Valkyrie is shown standing in front of a valve. In (b), the 
RViz display window shows the robot avatar and the point 
cloud data received from Valkyrie’s head-mounted Ensenso 
sensor. From this view, a supervisor can clearly identify the 
valve in the robot’s workspace, and can manually register the 
wheel affordance template to the location of the valve, seen 
in (c). Additionally, the wheel template allows the supervi- 
sor to adjust the hand pre-grasp, grasp, and turn-goal way- 
point locations (shown as different color hands, displayed 



Figure 2: Turning a valve using an affordance template. 


in the wheel reference frame). When the goal locations are 
finalized, they can be sent to the robot’s control system to 
accomplish the task (Figure [2|. 

Template parameters ( e.g . controller reference, object 
poses, navigational way-points) are meant to capture the key 
degrees of freedom of the task in an efficient representa- 
tion suitable for control over unreliable and/or unknown net- 
works. It is the fact that these parameters exist, however, that 
an affordance template is only a template. These parameters 
must be set in order to provide sufficient information for the 
task to be accomplished. As a result, affordance templates 
follow a methodology of shared control, which lets the su- 
pervisor and robot control different signals simultaneously, 
rather than on traded control, which assumes direct teleop- 
eration or full autonomy. A number of affordance templates 
were developed for use on the Valkyrie robot including tem- 
plates for door opening, walking, hose mating, valve turning, 
and ladder and stair climbing. 

Ongoing Work 

The Affordance Template framework follows a supervisory 
control paradigm of Plan, Teach, Monitor, Intervene, and 
Learn (PTMIL) ( [Sheridan 1992| . Currently, the framework 
supports only the PTMI steps of this paradigm as it only 
allows the supervisor to place a template in an appropriate 
location, adjust certain parameters, and execute the resulting 
plan to the robot (intervening if necessary). However, allow- 
ing the robot to learn from this experience is ongoing work. 
Currently, the framework is being extended along multiple 
dimensions. 

• Planning: Integration of motion planning tools such as 
Movelt! (http : / /move it .ros.org) will allow ex- 
tra levels of verification, intelligence, and visualization 
for the supervisor concerning the robot’s expected motion 
for accomplishing the goal(s) of the affordance template. 
Moreover, the ability to determine whether a waypoint 
goal is reachable after the supervisor adjusts that goal, 
could provide preliminary feedback on whether the task is 
accomplishable in the current context. More sophisticated 
uses of robot planning could also be used to eliminate the 
necessity for the supervisor to set all task parameters as is 
currently necessary, or to chain affordance templates to- 
gether to accomplish multiple extended behaviors (walk 
up to an object, pick it up, use it for some purpose, etc.) 

• Learning: Either in the absence of, or in conjunction 
with, more sophisticated planning techniques, affordance 


templates could monitor the parameter values over the 
course of multiple executions of a task and provide statis- 
tical feedback on whether those values are likely to lead 
to task success. The metric of success could be supervised 
(let the supervisor indicate task outcome after execution) 
or unsupervised depending on the task’s requirements. 

• Perceptual Registration: Rather than relying on the su- 
pervisor to register templates with sensor data (as in Fig- 
ure [If, autonomous point cloud registration techniques 
could be used. In fact, the robot could monitor its envi- 
ronment and provide “guesses” about likely affordances, 
“dropping in” corresponding affordance templates with 
pre-adjusted parameter settings into the virtual environ- 
ment. 

• Force-Based Tasks: Currently existing affordance tem- 
plates only allow the supervisor to set spatial parame- 
ters (object locations, end-effector waypoints, etc.). How- 
ever, force- or contact-based goals (apply a force along 
an axis of a certain magnitude, turn the wheel with a de- 
sired torque, etc.) could also be set in the framework. Such 
specifications will ultimately be necessary for guiding a 
robot to accomplish sophisticated manipulation tasks in 
real world industrial or service contexts. The visualization 
of these parameters in a 3D spatial environment is an in- 
teresting problem on its own terms, and currently various 
approaches are under investigation. 

Although some of these extensions individually may require 
significant research contributions, the affordance template 
framework provides an efficient representation for aggre- 
gating this functionality together in a unified structure. By 
keeping the supervisor “in the loop” and phasing in more au- 
tonomy as appropriate — whether in terms of planning, per- 
ception, or learning — the framework supports multiple lev- 
els of intervention that can ensure task success in multiple 
contexts. 
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